System and method for prioritizing AR information based on persistence of real-life objects in the user&#39;s view

ABSTRACT

Systems and methods described herein in accordance with some embodiments may display augmented reality (AR) information related to an object based on a predicted persistence of the object within view of the user. In some embodiments, display of AR information is prioritized based upon a prediction of how soon an object will disappear from view, perhaps if the predicted disappearance will occur sooner than some threshold time. Systems and methods described herein in accordance with some embodiments extend to mixed reality (MR) systems, and provide for displaying an image of an object that has become obstructed or has disappeared from view and/or for displaying AR information related to an object that has become obstructed or has disappeared from view.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a national stage application under 35 U.S.C.§ 371 of International Application No. PCT/US2018/032121, entitled“SYSTEM AND METHOD FOR PRIORITIZING AR INFORMATION BASED ON PERSISTENCEOF REAL-LIFE OBJECTS IN THE USER'S VIEW,” filed on May 10, 2018, whichclaims benefit under 35 U.S.C. § 119(e) from U.S. Provisional PatentApplication Ser. No. 62/510,099, filed May 23, 2017, entitled “Systemand Method for Prioritizing AR Information Based on Persistence ofReal-Life Objects in the User's View,” which is incorporated herein byreference in its entirety.

BACKGROUND

Augmented reality (AR) interfaces present information to a user relatingto some real-life (real-world) object that is present within the user'sview. Mixed reality (MR) interfaces present information to a user in away that combines virtual objects with a view of the real world. With MRdisplays, some of the objects are real, while some are virtual;additionally, some AR information about the real and/or virtual objectsmay also be displayed.

SUMMARY

Systems and methods described herein in accordance with some embodimentsmay display augmented reality (AR) information related to an objectbased on a predicted persistence of the object within view of the user.In some embodiments, display of AR information is prioritized based upona prediction of how soon an object will disappear from view, perhaps ifthe predicted disappearance will occur sooner than some threshold time.Systems and methods described herein in accordance with some embodimentsextend to mixed reality (MR) systems, and provide for displaying animage of an object that has become obstructed or has disappeared fromview and/or for displaying AR information related to an object that hasbecome obstructed or has disappeared from view.

In some embodiments, a method of prioritizing AR information based onobject persistence within a user's view may include: receiving sceneinformation; receiving a plurality of AR pairs, each AR pair comprisingan identification of an objects and AR information associated with theidentified object; determining persistence for objects in the pluralityof AR pairs; prioritizing display of AR information for objects based onthe determined persistence values for the corresponding objects; andsending information augmented with prioritize AR information to adisplay.

In some embodiments, determining persistence values may includedetermining how long each object may persist within a user's view, ordetermining an object's velocity and an object's visibility. Determiningan object's velocity may include performing motion tracking of theobject. Determining persistence values may include determining how longan object will persist in a user's view based on the object'scoordinates, the object's velocity, and boundaries of the received sceneinformation. In some embodiments, determining persistence values may usea straight line of motion model, or using historical motion to predictfuture motion. Persistence values may be determined up to thresholdprediction time limit, which in some embodiments may be 30 seconds. Insome embodiments, determining persistence values may include predictingpoints in a line of motion of an object and/or displaying the predictedpoints in the line of motion of the object. Visibility may be determinedfor the predicted points in the line of motion of the object, as part ofdetermining persistence values. In some embodiments, determiningpersistence values may include determining occlusion of an object. Insome embodiments, determining persistence for objects in the pluralityof AR pairs may include sending the plurality of AR pairs to apersistence determiner; and receiving the determined persistence forobjects in the plurality of AR pairs.

In some embodiments, prioritizing display of AR information may includepresenting AR information first for objects having a predictedpersistence shorter than a first threshold. In some embodiments,prioritizing display of AR information may include presenting ARinformation first for objects having a predicted persistence shorterthan a first threshold but longer than a second threshold. Prioritizingdisplay of AR information may include determining whether AR informationfor an object has been displayed; and responsive to determining that ARinformation for the object has been displayed, not displaying the ARinformation for the object again. Prioritizing display of AR informationmay include assigning a prioritization score to an AR pair; anddisplaying AR information corresponding to pairs having higherprioritization scores prior to displaying AR information correspondingto pairs having lower prioritization scores. Prioritizing display of ARinformation may include highlighting a display of the AR informationwith a different font size or color than AR information that is notprioritized. Some embodiments of the method may further include:responsive to receiving the scene information, sending the sceneinformation to an information service, and wherein receiving a pluralityof AR pairs includes receiving the plurality of AR pairs from theinformation service. Receiving scene information may include sensingscene information with a scene sensor. And in some embodiments, sceneinformation may include a point cloud.

In some embodiments, a method of prioritizing AR information based onobject persistence within a user's view may include tracking motion of aplurality of real-world objects; based on the tracking of the objects,predicting how long each object is likely to persist in the user's fieldof view; and prioritizing display of the AR information for thoseobjects having shorter predicted persistence, wherein prioritizingdisplay of the AR information comprises displaying first.

In some embodiments, a system may include a processor; and anon-transitory computer-readable medium storing instructions that areoperative, when executed by the processor, to perform the methodsdisclosed herein. A system my further include a display and a scenesensor.

BRIEF DESCRIPTION OF THE DRAWINGS

A more detailed understanding may be had from the following description,presented by way of example in conjunction with the accompanyingdrawings. Furthermore, like reference numerals in the figures indicatelike elements.

FIG. 1A is a system diagram illustrating an example communicationssystem in which one or more disclosed embodiments may be implemented.

FIG. 1B is a system diagram illustrating an example wirelesstransmit/receive unit (WTRU) that may be used within the communicationssystem illustrated in FIG. 1A according to some embodiments.

FIG. 1C is a system diagram illustrating an example radio access network(RAN) and an example core network (CN) that may be used within thecommunications system illustrated in FIG. 1A according to someembodiments.

FIG. 1D is a system diagram illustrating a further example RAN and afurther example CN that may be used within the communications systemillustrated in FIG. 1A according to some embodiments.

FIG. 1E illustrates a network entity that can perform methods disclosedherein, according to some embodiments.

FIG. 2 is a message flow diagram for a method of prioritizing augmentedreality (AR) information based on persistence of objects within theuser's view, according to some embodiments.

FIG. 3 is a message sequence diagram for a method of prioritizing ARinformation based on persistence of objects within the user's view,according to some embodiments.

FIGS. 4A-4B are a sequence of user views that show a basis for apersistence determination that may be used in some embodiments.

FIG. 5 is an illustration of a user view with AR information formultiple objects, as may be observed with some embodiments.

FIG. 6 is a message flow diagram for a method of prioritizing ARinformation and handling interruptions for an MR system, according tosome embodiments.

FIG. 7 is a message sequence diagram for a creating an MR presentation,possibly including AR information, and handling interruptions for an MRsystem, according to some embodiments.

FIGS. 8A-8D are illustrations of MR user views for actual and virtualobjects under various conditions, as may be observed with someembodiments.

FIGS. 9A-9D are illustrations showing example locations of user gazeswithin example AR information text.

FIGS. 10A-10C are illustrations of MR user views with a moving object,under various scenarios, as may be observed with some embodiments.

FIGS. 11A-11C are illustrations of MR user views involving calculationof displacement or placement of objects, as may be observed with someembodiments.

FIGS. 12A-12B are AR/MR user views for a scenario in which an object isobscured, as may be observed with some embodiments.

FIGS. 13A-13B are AR/MR user views for a scenario in which an objectmoves out of view, as may be observed with some embodiments.

FIGS. 14A-14E are illustrations of a heads-up AR display in variousscenarios, as may be observed with some embodiments.

The entities, connections, arrangements, and the like that are depictedin, and in connection with, the various figures are presented by way ofexample and not by way of limitation. As such, any and all statements orother indications as to what a particular figure depicts, what aparticular element or entity in a particular figure is or has, and anyand all similar statements, that may in isolation and out of context beread as absolute and therefore limiting, may only properly be read asbeing constructively preceded by a clause such as “In at least someembodiments, . . . ” For brevity and clarity of presentation, thisimplied leading clause is not repeated ad nauseum in the detaileddescription of the drawings.

DETAILED DESCRIPTION

A detailed description of illustrative embodiments will now be describedwith reference to the various Figures. Although this descriptionprovides a detailed example of possible implementations, it should benoted that the details are intended to be exemplary and in no way limitthe scope of the application.

Example Networks for Implementation of the Embodiments

FIG. 1A is a diagram illustrating an example communications system 100in which one or more disclosed embodiments may be implemented. Thecommunications system 100 may be a multiple access system that providescontent, such as voice, data, video, messaging, broadcast, etc., tomultiple wireless users. The communications system 100 may enablemultiple wireless users to access such content through the sharing ofsystem resources, including wireless bandwidth. For example, thecommunications systems 100 may employ one or more channel accessmethods, such as code division multiple access (CDMA), time divisionmultiple access (TDMA), frequency division multiple access (FDMA),orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tailunique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM(UW-OFDM), resource block-filtered OFDM, filter bank multicarrier(FBMC), and the like.

As shown in FIG. 1A, the communications system 100 may include wirelesstransmit/receive units (WTRUs) 102 a, 102 b, 102 c, 102 d, a RAN104/113, a CN 106/115, a public switched telephone network (PSTN) 108,the Internet 110, and other networks 112, though it will be appreciatedthat the disclosed embodiments contemplate any number of WTRUs, basestations, networks, and/or network elements. Each of the WTRUs 102 a,102 b, 102 c, 102 d may be any type of device configured to operateand/or communicate in a wireless environment. By way of example, theWTRUs 102 a, 102 b, 102 c, 102 d, any of which may be referred to as a“station” and/or a “STA”, may be configured to transmit and/or receivewireless signals and may include a user equipment (UE), a mobilestation, a fixed or mobile subscriber unit, a subscription-based unit, apager, a cellular telephone, a personal digital assistant (PDA), asmartphone, a laptop, a netbook, a personal computer, a wireless sensor,a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watchor other wearable, a head-mounted display (HMD), a vehicle, a drone, amedical device and applications (e.g., remote surgery), an industrialdevice and applications (e.g., a robot and/or other wireless devicesoperating in an industrial and/or an automated processing chaincontexts), a consumer electronics device, a device operating oncommercial and/or industrial wireless networks, and the like. By way ofexample, the client device 102 a is depicted as a cellular telephone;the client device 102 b is depicted as an HMD for 3D augmented reality(AR), virtual reality (VR) and/or mixed reality (MR) display; clientdevice 102 c is depicted as a desktop compute; and the client device 102d is depicted as a tablet computer. Any of WTRUs 102 a, 102 b, 102 c and102 d may be interchangeably referred to as a UE.

The communications systems 100 may also include a base station 114 aand/or a base station 114 b. Each of the base stations 114 a, 114 b maybe any type of device configured to wirelessly interface with at leastone of the WTRUs 102 a, 102 b, 102 c, 102 d to facilitate access to oneor more communication networks, such as the CN 106/115, the Internet110, and/or the other networks 112. By way of example, the base stations114 a, 114 b may be a base transceiver station (BTS), a Node-B, an eNodeB, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller,an access point (AP), a wireless router, and the like. While the basestations 114 a, 114 b are each depicted as a single element, it will beappreciated that the base stations 114 a, 114 b may include any numberof interconnected base stations and/or network elements.

The base station 114 a may be part of the RAN 104/113, which may alsoinclude other base stations and/or network elements (not shown), such asa base station controller (BSC), a radio network controller (RNC), relaynodes, etc. The base station 114 a and/or the base station 114 b may beconfigured to transmit and/or receive wireless signals on one or morecarrier frequencies, which may be referred to as a cell (not shown).These frequencies may be in licensed spectrum, unlicensed spectrum, or acombination of licensed and unlicensed spectrum. A cell may providecoverage for a wireless service to a specific geographical area that maybe relatively fixed or that may change over time. The cell may furtherbe divided into cell sectors. For example, the cell associated with thebase station 114 a may be divided into three sectors. Thus, in oneembodiment, the base station 114 a may include three transceivers, i.e.,one for each sector of the cell. In an embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and mayutilize multiple transceivers for each sector of the cell. For example,beamforming may be used to transmit and/or receive signals in desiredspatial directions.

The base stations 114 a, 114 b may communicate with one or more of theWTRUs 102 a, 102 b, 102 c, 102 d over an air interface 116, which may beany suitable wireless communication link (e.g., radio frequency (RF),microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet(UV), visible light, etc.). The air interface 116 may be establishedusing any suitable radio access technology (RAT).

More specifically, as noted above, the communications system 100 may bea multiple access system and may employ one or more channel accessschemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. Forexample, the base station 114 a in the RAN 104/113 and the WTRUs 102 a,102 b, 102 c may implement a radio technology such as Universal MobileTelecommunications System (UMTS) Terrestrial Radio Access (UTRA), whichmay establish the air interface 115/116/117 using wideband CDMA (WCDMA).WCDMA may include communication protocols such as High-Speed PacketAccess (HSPA) and/or Evolved HSPA (HSPA+). HSPA may include High-SpeedDownlink (DL) Packet Access (HSDPA) and/or High-Speed UL Packet Access(HSUPA).

In an embodiment, the base station 114 a and the WTRUs 102 a, 102 b, 102c may implement a radio technology such as Evolved UMTS TerrestrialRadio Access (E-UTRA), which may establish the air interface 116 usingLong Term Evolution (LTE) and/or LTE-Advanced (LTE-A) and/orLTE-Advanced Pro (LTE-A Pro). In an embodiment, the base station 114 aand the WTRUs 102 a, 102 b, 102 c may implement a radio technology suchas NR Radio Access, which may establish the air interface 116 using NewRadio (NR).

In an embodiment, the base station 114 a and the WTRUs 102 a, 102 b, 102c may implement multiple radio access technologies. For example, thebase station 114 a and the WTRUs 102 a, 102 b, 102 c may implement LTEradio access and NR radio access together, for instance using dualconnectivity (DC) principles. Thus, the air interface utilized by WTRUs102 a, 102 b, 102 c may be characterized by multiple types of radioaccess technologies and/or transmissions sent to/from multiple types ofbase stations (e.g., a eNB and a gNB).

In other embodiments, the base station 114 a and the WTRUs 102 a, 102 b,102 c may implement radio technologies such as IEEE 802.11 (i.e.,Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperabilityfor Microwave Access (WiMAX)), CDMA2000, CDMA2000 1×, CDMA2000 EV-DO,Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), InterimStandard 856 (IS-856), Global System for Mobile communications (GSM),Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and thelike.

The base station 114 b in FIG. 1A may be a wireless router, Home Node B,Home eNode B, or access point, for example, and may utilize any suitableRAT for facilitating wireless connectivity in a localized area, such asa place of business, a home, a vehicle, a campus, an industrialfacility, an air corridor (e.g., for use by drones), a roadway, and thelike. In one embodiment, the base station 114 b and the WTRUs 102 c, 102d may implement a radio technology such as IEEE 802.11 to establish awireless local area network (WLAN). In an embodiment, the base station114 b and the WTRUs 102 c, 102 d may implement a radio technology suchas IEEE 802.15 to establish a wireless personal area network (WPAN). Inyet another embodiment, the base station 114 b and the WTRUs 102 c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE,LTE-A, LTE-A Pro, NR etc.) to establish a picocell or femtocell. Asshown in FIG. 1A, the base station 114 b may have a direct connection tothe Internet 110. Thus, the base station 114 b may not be required toaccess the Internet 110 via the CN 106/115.

The RAN 104/113 may be in communication with the CN 106/115, which maybe any type of network configured to provide voice, data, applications,and/or voice over internet protocol (VoIP) services to one or more ofthe WTRUs 102 a, 102 b, 102 c, 102 d. The data may have varying qualityof service (QoS) requirements, such as differing throughputrequirements, latency requirements, error tolerance requirements,reliability requirements, data throughput requirements, mobilityrequirements, and the like. The CN 106/115 may provide call control,billing services, mobile location-based services, pre-paid calling,Internet connectivity, video distribution, etc., and/or performhigh-level security functions, such as user authentication. Although notshown in FIG. 1A, it will be appreciated that the RAN 104/113 and/or theCN 106/115 may be in direct or indirect communication with other RANsthat employ the same RAT as the RAN 104/113 or a different RAT. Forexample, in addition to being connected to the RAN 104/113, which may beutilizing a NR radio technology, the CN 106/115 may also be incommunication with another RAN (not shown) employing a GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

The CN 106/115 may also serve as a gateway for the WTRUs 102 a, 102 b,102 c, 102 d to access the PSTN 108, the Internet 110, and/or the othernetworks 112. The PSTN 108 may include circuit-switched telephonenetworks that provide plain old telephone service (POTS). The Internet110 may include a global system of interconnected computer networks anddevices that use common communication protocols, such as thetransmission control protocol (TCP), user datagram protocol (UDP) and/orthe internet protocol (IP) in the TCP/IP internet protocol suite. Thenetworks 112 may include wired and/or wireless communications networksowned and/or operated by other service providers. For example, thenetworks 112 may include another CN connected to one or more RANs, whichmay employ the same RAT as the RAN 104/113 or a different RAT.

Some or all of the WTRUs 102 a, 102 b, 102 c, 102 d in thecommunications system 100 may include multi-mode capabilities (e.g., theWTRUs 102 a, 102 b, 102 c, 102 d may include multiple transceivers forcommunicating with different wireless networks over different wirelesslinks). For example, the WTRU 102 c shown in FIG. 1A may be configuredto communicate with the base station 114 a, which may employ acellular-based radio technology, and with the base station 114 b, whichmay employ an IEEE 802 radio technology.

FIG. 1B is a system diagram illustrating an example WTRU 102. As shownin FIG. 1B, the WTRU 102 may include a processor 118, a transceiver 120,a transmit/receive element 122, a speaker/microphone 124, a keypad 126,a display/touchpad 128, non-removable memory 130, removable memory 132,a power source 134, a global positioning system (GPS) chipset 136,and/or other peripherals 138, among others. It will be appreciated thatthe WTRU 102 may include any sub-combination of the foregoing elementswhile remaining consistent with an embodiment.

The processor 118 may be a general purpose processor, a special purposeprocessor, a conventional processor, a digital signal processor (DSP), aplurality of microprocessors, one or more microprocessors in associationwith a DSP core, a controller, a microcontroller, Application SpecificIntegrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs)circuits, any other type of integrated circuit (IC), a state machine,and the like. The processor 118 may perform signal coding, dataprocessing, power control, input/output processing, and/or any otherfunctionality that enables the WTRU 102 to operate in a wirelessenvironment. The processor 118 may be coupled to the transceiver 120,which may be coupled to the transmit/receive element 122. While FIG. 1Bdepicts the processor 118 and the transceiver 120 as separatecomponents, it will be appreciated that the processor 118 and thetransceiver 120 may be integrated together in an electronic package orchip.

The transmit/receive element 122 may be configured to transmit signalsto, or receive signals from, a base station (e.g., the base station 114a) over the air interface 116. For example, in one embodiment, thetransmit/receive element 122 may be an antenna configured to transmitand/or receive RF signals. In an embodiment, the transmit/receiveelement 122 may be an emitter/detector configured to transmit and/orreceive IR, UV, or visible light signals, for example. In yet anotherembodiment, the transmit/receive element 122 may be configured totransmit and/or receive both RF and light signals. It will beappreciated that the transmit/receive element 122 may be configured totransmit and/or receive any combination of wireless signals.

Although the transmit/receive element 122 is depicted in FIG. 1B as asingle element, the WTRU 102 may include any number of transmit/receiveelements 122. More specifically, the WTRU 102 may employ MIMOtechnology. Thus, in one embodiment, the WTRU 102 may include two ormore transmit/receive elements 122 (e.g., multiple antennas) fortransmitting and receiving wireless signals over the air interface 116.

The transceiver 120 may be configured to modulate the signals that areto be transmitted by the transmit/receive element 122 and to demodulatethe signals that are received by the transmit/receive element 122. Asnoted above, the WTRU 102 may have multi-mode capabilities. Thus, thetransceiver 120 may include multiple transceivers for enabling the WTRU102 to communicate via multiple RATs, such as NR and IEEE 802.11, forexample.

The processor 118 of the WTRU 102 may be coupled to, and may receiveuser input data from, the speaker/microphone 124, the keypad 126, and/orthe display/touchpad 128 (e.g., a liquid crystal display (LCD) displayunit or organic light-emitting diode (OLED) display unit). The processor118 may also output user data to the speaker/microphone 124, the keypad126, and/or the display/touchpad 128. In addition, the processor 118 mayaccess information from, and store data in, any type of suitable memory,such as the non-removable memory 130 and/or the removable memory 132.The non-removable memory 130 may include random-access memory (RAM),read-only memory (ROM), a hard disk, or any other type of memory storagedevice. The removable memory 132 may include a subscriber identitymodule (SIM) card, a memory stick, a secure digital (SD) memory card,and the like. In other embodiments, the processor 118 may accessinformation from, and store data in, memory that is not physicallylocated on the WTRU 102, such as on a server or a home computer (notshown).

The processor 118 may receive power from the power source 134, and maybe configured to distribute and/or control the power to the othercomponents in the WTRU 102. The power source 134 may be any suitabledevice for powering the WTRU 102. For example, the power source 134 mayinclude one or more dry cell batteries (e.g., nickel-cadmium (NiCd),nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion),etc.), solar cells, fuel cells, and the like.

The processor 118 may also be coupled to the GPS chipset 136, which maybe configured to provide location information (e.g., longitude andlatitude) regarding the current location of the WTRU 102. In additionto, or in lieu of, the information from the GPS chipset 136, the WTRU102 may receive location information over the air interface 116 from abase station (e.g., base stations 114 a, 114 b) and/or determine itslocation based on the timing of the signals being received from two ormore nearby base stations. It will be appreciated that the WTRU 102 mayacquire location information by way of any suitablelocation-determination method while remaining consistent with anembodiment.

The processor 118 may further be coupled to other peripherals 138, whichmay include one or more software and/or hardware modules that provideadditional features, functionality and/or wired or wirelessconnectivity. For example, the peripherals 138 may include anaccelerometer, an e-compass, a satellite transceiver, a digital camera(for photographs and/or video), a universal serial bus (USB) port, avibration device, a television transceiver, a hands free headset, aBluetooth® module, a frequency modulated (FM) radio unit, a digitalmusic player, a media player, a video game player module, an Internetbrowser, an Augmented Reality, Virtual Reality, and/or Mixed Reality(AR/VR/MR) device, an activity tracker, and the like. The peripherals138 may include one or more sensors, the sensors may be one or more of agyroscope, an accelerometer, a hall effect sensor, a magnetometer, anorientation sensor, a proximity sensor, a temperature sensor, a timesensor; a geolocation sensor; an altimeter, a light sensor, a touchsensor, a magnetometer, a barometer, a gesture sensor, a biometricsensor, and/or a humidity sensor.

The WTRU 102 may include a full duplex radio for which transmission andreception of some or all of the signals (e.g., associated withparticular subframes for both the UL (e.g., for transmission) anddownlink (e.g., for reception) may be concurrent and/or simultaneous.The full duplex radio may include an interference management unit toreduce and or substantially eliminate self-interference via eitherhardware (e.g., a choke) or signal processing via a processor (e.g., aseparate processor (not shown) or via processor 118). In an embodiment,the WRTU 102 may include a half-duplex radio for which transmission andreception of some or all of the signals (e.g., associated withparticular subframes for either the UL (e.g., for transmission) or thedownlink (e.g., for reception)).

FIG. 1C is a system diagram illustrating the RAN 104 and the CN 106according to an embodiment. As noted above, the RAN 104 may employ anE-UTRA radio technology to communicate with the WTRUs 102 a, 102 b, 102c over the air interface 116. The RAN 104 may also be in communicationwith the CN 106.

The RAN 104 may include eNode-Bs 160 a, 160 b, 160 c, though it will beappreciated that the RAN 104 may include any number of eNode-Bs whileremaining consistent with an embodiment. The eNode-Bs 160 a, 160 b, 160c may each include one or more transceivers for communicating with theWTRUs 102 a, 102 b, 102 c over the air interface 116. In one embodiment,the eNode-Bs 160 a, 160 b, 160 c may implement MIMO technology. Thus,the eNode-B 160 a, for example, may use multiple antennas to transmitwireless signals to, and/or receive wireless signals from, the WTRU 102a.

Each of the eNode-Bs 160 a, 160 b, 160 c may be associated with aparticular cell (not shown) and may be configured to handle radioresource management decisions, handover decisions, scheduling of usersin the UL and/or DL, and the like. As shown in FIG. 1C, the eNode-Bs 160a, 160 b, 160 c may communicate with one another over an X2 interface.

The CN 106 shown in FIG. 1C may include a mobility management entity(MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN)gateway (or PGW) 166. While each of the foregoing elements are depictedas part of the CN 106, it will be appreciated that any of these elementsmay be owned and/or operated by an entity other than the CN operator.

The MME 162 may be connected to each of the eNode-Bs 162 a, 162 b, 162 cin the RAN 104 via an S1 interface and may serve as a control node. Forexample, the MME 162 may be responsible for authenticating users of theWTRUs 102 a, 102 b, 102 c, bearer activation/deactivation, selecting aparticular serving gateway during an initial attach of the WTRUs 102 a,102 b, 102 c, and the like. The MME 162 may provide a control planefunction for switching between the RAN 104 and other RANs (not shown)that employ other radio technologies, such as GSM and/or WCDMA.

The SGW 164 may be connected to each of the eNode Bs 160 a, 160 b, 160 cin the RAN 104 via the S1 interface. The SGW 164 may generally route andforward user data packets to/from the WTRUs 102 a, 102 b, 102 c. The SGW164 may perform other functions, such as anchoring user planes duringinter-eNode B handovers, triggering paging when DL data is available forthe WTRUs 102 a, 102 b, 102 c, managing and storing contexts of theWTRUs 102 a, 102 b, 102 c, and the like.

The SGW 164 may be connected to the PGW 166, which may provide the WTRUs102 a, 102 b, 102 c with access to packet-switched networks, such as theInternet 110, to facilitate communications between the WTRUs 102 a, 102b, 102 c and IP-enabled devices.

The CN 106 may facilitate communications with other networks. Forexample, the CN 106 may provide the WTRUs 102 a, 102 b, 102 c withaccess to circuit-switched networks, such as the PSTN 108, to facilitatecommunications between the WTRUs 102 a, 102 b, 102 c and traditionalland-line communications devices. For example, the CN 106 may include,or may communicate with, an IP gateway (e.g., an IP multimedia subsystemserver) that serves as an interface between the CN 106 and the PSTN 108.In addition, the CN 106 may provide the WTRUs 102 a, 102 b, 102 c withaccess to the other networks 112, which may include other wired and/orwireless networks that are owned and/or operated by other serviceproviders.

Although the WTRU is described in FIGS. 1A-1D as a wireless terminal, itis contemplated that in certain representative embodiments that such aterminal may use (e.g., temporarily or permanently) wired communicationinterfaces with the communication network. In representativeembodiments, the other network 112 may be a WLAN.

A WLAN in Infrastructure Basic Service Set (BSS) mode may have an AccessPoint (AP) for the BSS and one or more stations (STAs) associated withthe AP. The AP may have an access or an interface to a DistributionSystem (DS) or another type of wired/wireless network that carriestraffic in to and/or out of the BSS. Traffic to STAs that originatesfrom outside the BSS may arrive through the AP and may be delivered tothe STAs. Traffic originating from STAs to destinations outside the BSSmay be sent to the AP to be delivered to respective destinations.Traffic between STAs within the BSS may be sent through the AP, forexample, where the source STA may send traffic to the AP and the AP maydeliver the traffic to the destination STA. The traffic between STAswithin a BSS may be considered and/or referred to as peer-to-peertraffic. The peer-to-peer traffic may be sent between (e.g., directlybetween) the source and destination STAs with a direct link setup (DLS).In certain representative embodiments, the DLS may use an 802.11e DLS oran 802.11z tunneled DLS (TDLS). A WLAN using an Independent BSS (IBSS)mode may not have an AP, and the STAs (e.g., all of the STAs) within orusing the IBSS may communicate directly with each other. The IBSS modeof communication may sometimes be referred to herein as an “ad-hoc” modeof communication.

When using the 802.11ac infrastructure mode of operation or a similarmode of operations, the AP may transmit a beacon on a fixed channel,such as a primary channel. The primary channel may be a fixed width(e.g., 20 MHz wide bandwidth) or a dynamically set width via signaling.The primary channel may be the operating channel of the BSS and may beused by the STAs to establish a connection with the AP. In certainrepresentative embodiments, Carrier Sense Multiple Access with CollisionAvoidance (CSMA/CA) may be implemented, for example in in 802.11systems. For CSMA/CA, the STAs (e.g., every STA), including the AP, maysense the primary channel. If the primary channel is sensed/detectedand/or determined to be busy by a particular STA, the particular STA mayback off. One STA (e.g., only one station) may transmit at any giventime in a given BSS.

High Throughput (HT) STAs may use a 40 MHz wide channel forcommunication, for example, via a combination of the primary 20 MHzchannel with an adjacent or nonadjacent 20 MHz channel to form a 40 MHzwide channel.

Very High Throughput (VHT) STAs may support 20 MHz, 40 MHz, 80 MHz,and/or 160 MHz wide channels. The 40 MHz, and/or 80 MHz, channels may beformed by combining contiguous 20 MHz channels. A 160 MHz channel may beformed by combining 8 contiguous 20 MHz channels, or by combining twonon-contiguous 80 MHz channels, which may be referred to as an 80+80configuration. For the 80+80 configuration, the data, after channelencoding, may be passed through a segment parser that may divide thedata into two streams. Inverse Fast Fourier Transform (IFFT) processing,and time domain processing, may be done on each stream separately. Thestreams may be mapped on to the two 80 MHz channels, and the data may betransmitted by a transmitting STA. At the receiver of the receiving STA,the above described operation for the 80+80 configuration may bereversed, and the combined data may be sent to the Medium Access Control(MAC).

Sub 1 GHz modes of operation are supported by 802.11af and 802.11ah. Thechannel operating bandwidths, and carriers, are reduced in 802.11af and802.11ah relative to those used in 802.11n, and 802.11ac. 802.11afsupports 5 MHz, 10 MHz and 20 MHz bandwidths in the TV White Space(TVWS) spectrum, and 802.11ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and16 MHz bandwidths using non-TVWS spectrum. According to a representativeembodiment, 802.11ah may support Meter Type Control/Machine-TypeCommunications, such as MTC devices in a macro coverage area. MTCdevices may have certain capabilities, for example, limited capabilitiesincluding support for (e.g., only support for) certain and/or limitedbandwidths. The MTC devices may include a battery with a battery lifeabove a threshold (e.g., to maintain a very long battery life).

WLAN systems, which may support multiple channels, and channelbandwidths, such as 802.11n, 802.11ac, 802.11af, and 802.11ah, include achannel which may be designated as the primary channel. The primarychannel may have a bandwidth equal to the largest common operatingbandwidth supported by all STAs in the BSS. The bandwidth of the primarychannel may be set and/or limited by a STA, from among all STAs inoperating in a BSS, which supports the smallest bandwidth operatingmode. In the example of 802.11ah, the primary channel may be 1 MHz widefor STAs (e.g., MTC type devices) that support (e.g., only support) a 1MHz mode, even if the AP, and other STAs in the BSS support 2 MHz, 4MHz, 8 MHz, 16 MHz, and/or other channel bandwidth operating modes.Carrier sensing and/or Network Allocation Vector (NAV) settings maydepend on the status of the primary channel. If the primary channel isbusy, for example, due to a STA (which supports only a 1 MHz operatingmode), transmitting to the AP, the entire available frequency bands maybe considered busy even though a majority of the frequency bands remainsidle and may be available.

In the United States, the available frequency bands, which may be usedby 802.11ah, are from 902 MHz to 928 MHz. In Korea, the availablefrequency bands are from 917.5 MHz to 923.5 MHz. In Japan, the availablefrequency bands are from 916.5 MHz to 927.5 MHz. The total bandwidthavailable for 802.11ah is 6 MHz to 26 MHz depending on the country code.

FIG. 1D is a system diagram illustrating the RAN 113 and the CN 115according to an embodiment. As noted above, the RAN 113 may employ an NRradio technology to communicate with the WTRUs 102 a, 102 b, 102 c overthe air interface 116. The RAN 113 may also be in communication with theCN 115.

The RAN 113 may include gNBs 180 a, 180 b, 180 c, though it will beappreciated that the RAN 113 may include any number of gNBs whileremaining consistent with an embodiment. The gNBs 180 a, 180 b, 180 cmay each include one or more transceivers for communicating with theWTRUs 102 a, 102 b, 102 c over the air interface 116. In one embodiment,the gNBs 180 a, 180 b, 180 c may implement MIMO technology. For example,gNBs 180 a, 108 b may utilize beamforming to transmit signals to and/orreceive signals from the gNBs 180 a, 180 b, 180 c. Thus, the gNB 180 a,for example, may use multiple antennas to transmit wireless signals to,and/or receive wireless signals from, the WTRU 102 a. In an embodiment,the gNBs 180 a, 180 b, 180 c may implement carrier aggregationtechnology. For example, the gNB 180 a may transmit multiple componentcarriers to the WTRU 102 a (not shown). A subset of these componentcarriers may be on unlicensed spectrum while the remaining componentcarriers may be on licensed spectrum. In an embodiment, the gNBs 180 a,180 b, 180 c may implement Coordinated Multi-Point (CoMP) technology.For example, WTRU 102 a may receive coordinated transmissions from gNB180 a and gNB 180 b (and/or gNB 180 c).

The WTRUs 102 a, 102 b, 102 c may communicate with gNBs 180 a, 180 b,180 c using transmissions associated with a scalable numerology. Forexample, the OFDM symbol spacing and/or OFDM subcarrier spacing may varyfor different transmissions, different cells, and/or different portionsof the wireless transmission spectrum. The WTRUs 102 a, 102 b, 102 c maycommunicate with gNBs 180 a, 180 b, 180 c using subframe or transmissiontime intervals (TTIs) of various or scalable lengths (e.g., containingvarying number of OFDM symbols and/or lasting varying lengths ofabsolute time).

The gNBs 180 a, 180 b, 180 c may be configured to communicate with theWTRUs 102 a, 102 b, 102 c in a standalone configuration and/or anon-standalone configuration. In the standalone configuration, WTRUs 102a, 102 b, 102 c may communicate with gNBs 180 a, 180 b, 180 c withoutalso accessing other RANs (e.g., such as eNode-Bs 160 a, 160 b, 160 c).In the standalone configuration, WTRUs 102 a, 102 b, 102 c may utilizeone or more of gNBs 180 a, 180 b, 180 c as a mobility anchor point. Inthe standalone configuration, WTRUs 102 a, 102 b, 102 c may communicatewith gNBs 180 a, 180 b, 180 c using signals in an unlicensed band. In anon-standalone configuration WTRUs 102 a, 102 b, 102 c may communicatewith/connect to gNBs 180 a, 180 b, 180 c while also communicatingwith/connecting to another RAN such as eNode-Bs 160 a, 160 b, 160 c. Forexample, WTRUs 102 a, 102 b, 102 c may implement DC principles tocommunicate with one or more gNBs 180 a, 180 b, 180 c and one or moreeNode-Bs 160 a, 160 b, 160 c substantially simultaneously. In thenon-standalone configuration, eNode-Bs 160 a, 160 b, 160 c may serve asa mobility anchor for WTRUs 102 a, 102 b, 102 c and gNBs 180 a, 180 b,180 c may provide additional coverage and/or throughput for servicingWTRUs 102 a, 102 b, 102 c.

Each of the gNBs 180 a, 180 b, 180 c may be associated with a particularcell (not shown) and may be configured to handle radio resourcemanagement decisions, handover decisions, scheduling of users in the ULand/or DL, support of network slicing, dual connectivity, interworkingbetween NR and E-UTRA, routing of user plane data towards User PlaneFunction (UPF) 184 a, 184 b, routing of control plane informationtowards Access and Mobility Management Function (AMF) 182 a, 182 b andthe like. As shown in FIG. 1D, the gNBs 180 a, 180 b, 180 c maycommunicate with one another over an Xn interface.

The CN 115 shown in FIG. 1D may include at least one AMF 182 a, 182 b,at least one UPF 184 a,184 b, at least one Session Management Function(SMF) 183 a, 183 b, and possibly a Data Network (DN) 185 a, 185 b. Whileeach of the foregoing elements are depicted as part of the CN 115, itwill be appreciated that any of these elements may be owned and/oroperated by an entity other than the CN operator.

The AMF 182 a, 182 b may be connected to one or more of the gNBs 180 a,180 b, 180 c in the RAN 113 via an N2 interface and may serve as acontrol node. For example, the AMF 182 a, 182 b may be responsible forauthenticating users of the WTRUs 102 a, 102 b, 102 c, support fornetwork slicing (e.g., handling of different PDU sessions with differentrequirements), selecting a particular SMF 183 a, 183 b, management ofthe registration area, termination of NAS signaling, mobilitymanagement, and the like. Network slicing may be used by the AMF 182 a,182 b in order to customize CN support for WTRUs 102 a, 102 b, 102 cbased on the types of services being utilized WTRUs 102 a, 102 b, 102 c.For example, different network slices may be established for differentuse cases such as services relying on ultra-reliable low latency (URLLC)access, services relying on enhanced massive mobile broadband (eMBB)access, services for machine type communication (MTC) access, and/or thelike. The AMF 162 may provide a control plane function for switchingbetween the RAN 113 and other RANs (not shown) that employ other radiotechnologies, such as LTE, LTE-A, LTE-A Pro, and/or non-3GPP accesstechnologies such as WiFi.

The SMF 183 a, 183 b may be connected to an AMF 182 a, 182 b in the CN115 via an N11 interface. The SMF 183 a, 183 b may also be connected toa UPF 184 a, 184 b in the CN 115 via an N4 interface. The SMF 183 a, 183b may select and control the UPF 184 a, 184 b and configure the routingof traffic through the UPF 184 a, 184 b. The SMF 183 a, 183 b mayperform other functions, such as managing and allocating UE IP address,managing PDU sessions, controlling policy enforcement and QoS, providingdownlink data notifications, and the like. A PDU session type may beIP-based, non-IP based, Ethernet-based, and the like.

The UPF 184 a, 184 b may be connected to one or more of the gNBs 180 a,180 b, 180 c in the RAN 113 via an N3 interface, which may provide theWTRUs 102 a, 102 b, 102 c with access to packet-switched networks, suchas the Internet 110, to facilitate communications between the WTRUs 102a, 102 b, 102 c and IP-enabled devices. The UPF 184, 184 b may performother functions, such as routing and forwarding packets, enforcing userplane policies, supporting multi-homed PDU sessions, handling user planeQoS, buffering downlink packets, providing mobility anchoring, and thelike.

The CN 115 may facilitate communications with other networks. Forexample, the CN 115 may include, or may communicate with, an IP gateway(e.g., an IP multimedia subsystem server) that serves as an interfacebetween the CN 115 and the PSTN 108. In addition, the CN 115 may providethe WTRUs 102 a, 102 b, 102 c with access to the other networks 112,which may include other wired and/or wireless networks that are ownedand/or operated by other service providers. In one embodiment, the WTRUs102 a, 102 b, 102 c may be connected to a local Data Network (DN) 185 a,185 b through the UPF 184 a, 184 b via the N3 interface to the UPF 184a, 184 b and an N6 interface between the UPF 184 a, 184 b and the DN 185a, 185 b.

FIG. 1E depicts an exemplary network entity 190 that may be used inembodiments of the present disclosure, for example as an AR or MRmanager or as a UE. As depicted in FIG. 1E, network entity 190 includesa communication interface 192, a processor 194, and non-transitory datastorage 196, all of which are communicatively linked by a bus, network,or other communication path 198.

Communication interface 192 may include one or more wired communicationinterfaces and/or one or more wireless-communication interfaces. Withrespect to wired communication, communication interface 192 may includeone or more interfaces such as Ethernet interfaces, as an example. Withrespect to wireless communication, communication interface 192 mayinclude components such as one or more antennae, one or moretransceivers/chipsets designed and configured for one or more types ofwireless (e.g., LTE) communication, and/or any other components deemedsuitable by those of skill in the relevant art. And further with respectto wireless communication, communication interface 192 may be equippedat a scale and with a configuration appropriate for acting on thenetwork side—as opposed to the client side—of wireless communications(e.g., LTE communications, Wi-Fi communications, and the like). Thus,communication interface 192 may include the appropriate equipment andcircuitry (perhaps including multiple transceivers) for serving multiplemobile stations, UEs, or other access terminals in a coverage area.

Processor 194 may include one or more processors of any type deemedsuitable by those of skill in the relevant art, some examples includinga general-purpose microprocessor, a graphics processing unit (GPU) and adedicated digital signal processor (DSP). Data storage 196 may take theform of any non-transitory computer-readable medium or combination ofsuch media, some examples including flash memory, read-only memory(ROM), and random-access memory (RAM) to name but a few, as any one ormore types of non-transitory data storage deemed suitable by those ofskill in the relevant art could be used. As depicted in FIG. 1E, datastorage 196 is a non-transitory computer-readable media that containsprogram instructions 197 executable by processor 194 for carrying outvarious combinations of the various network-entity functions describedherein. Processor 194 is configured to execute instructions 197 storedin memory 196.

In view of FIGS. 1A-1E, and the corresponding description of FIGS.1A-1E, one or more, or all, of the functions described herein withregard to one or more of: WTRU 102 a-d, Base Station 114 a-b, eNode-B160 a-c, MME 162, SGW 164, PGW 166, gNB 180 a-c, AMF 182 a-b, UPF 184a-b, SMF 183 a-b, DN 185 a-b, and/or any other device(s) describedherein, may be performed by one or more emulation devices (not shown).The emulation devices may be one or more devices configured to emulateone or more, or all, of the functions described herein. For example, theemulation devices may be used to test other devices and/or to simulatenetwork and/or WTRU functions. By way of example, the client device 102a is depicted as a cellular telephone; the client device 102 b isdepicted as an HMD for 3D AR, VR and/or MR display; client device 102 cis depicted as a desktop compute; and the client device 102 d isdepicted as a tablet computer. It should be understood that any of thesedevices, or other devices having similar functionality, may be suitablefor either the client side device or the server side device.

The emulation devices may be designed to implement one or more tests ofother devices in a lab environment and/or in an operator networkenvironment. For example, the one or more emulation devices may performthe one or more, or all, functions while being fully or partiallyimplemented and/or deployed as part of a wired and/or wirelesscommunication network in order to test other devices within thecommunication network. The one or more emulation devices may perform theone or more, or all, functions while being temporarilyimplemented/deployed as part of a wired and/or wireless communicationnetwork. The emulation device may be directly coupled to another devicefor purposes of testing and/or may performing testing using over-the-airwireless communications.

The one or more emulation devices may perform the one or more, includingall, functions while not being implemented/deployed as part of a wiredand/or wireless communication network. For example, the emulationdevices may be utilized in a testing scenario in a testing laboratoryand/or a non-deployed (e.g., testing) wired and/or wirelesscommunication network in order to implement testing of one or morecomponents. The one or more emulation devices may be test equipment.Direct RF coupling and/or wireless communications via RF circuitry(e.g., which may include one or more antennas) may be used by theemulation devices to transmit and/or receive data.

Presentation of AR and MR Information.

Systems and methods described herein in accordance with some embodimentsmay display augmented reality (AR) information related to an objectbased on a predicted persistence of the object within view of the user.In some embodiments, display of AR information is prioritized based upona prediction of how soon an object will disappear from view, perhaps ifthe predicted disappearance will occur sooner than some threshold time.Systems and methods described herein in accordance with some embodimentsextend to mixed reality (MR) systems, and provide for displaying animage of an object that has become obstructed or has disappeared fromview and/or for displaying AR information related to an object that hasbecome obstructed or has disappeared from view.

AR interfaces present information to a user relating to some real-life(real-world) object that is present within the user's view. Such systemsmay have challenges with ordering or prioritizing presentation of ARinformation, and may risk overloading a user or not presenting theinformation that is most acutely needed. MR interfaces presentinformation to a user in a way that combines virtual objects with a viewof the real world. With MR displays, some of the objects are real, whilesome are virtual; additionally, some AR information about the realand/or virtual objects may also be displayed. MR systems may presentsimilar challenges regarding ordering or prioritizing presentation of ARinformation, and may thus likewise may present a risk of overloading auser or not presenting the information that is most acutely needed.Further, both AR and MR systems may also have challenges presenting ARinformation for an object when there has been an interruption of auser's interaction with or view of the object.

Certain systems may have challenges with presenting AR information forreal-world objects to a user in a manner that maximizes the user'sopportunity to benefit from available AR information. For example, thesesystems may have challenges when there is relative motion between theuser and the object that results in the real-world object disappearingfrom view. Certain AR systems do not account for how long an object willlikely to remain visible to a user. Thus, the user may be viewing ARinformation for an object that will persist in the user's view, even asanother object is soon to disappear. Thus, the user may miss anopportunity to receive AR information about the object that does notpersist in view.

In some embodiments of systems and methods described herein, ARinformation for real-world objects is presented such that real-worldobjects that are determined to be most likely to disappear soonest maybe presented first, or presented with emphasis, such as highlighting.That is, AR information for objects likely to disappear from view may bepresented prior to presenting AR information for other objects that arelikely to remain in view, AR information both types of objects may bepresented simultaneously with highlighting for objects likely todisappear. In some embodiments, there may be a combination.

Systems and methods disclosed herein in accordance with some embodimentsmay prioritize AR information based on the persistence of an objectwithin the user's view. For some embodiments, methods may includereceiving a plurality of one or more AR pairs, e.g., a list of one ormore AR pairs, where each AR pair comprises a real-world object andassociated AR information; determining the persistence of objects; andpresenting AR information based on the persistence of objects in theuser's view, such that AR information is prioritized for objects with apersistence shorter than a threshold. As used herein, in accordance withsome embodiments, a pair is a real-world (real-life) object (ordescription of a location of a real-world object) and associated ARinformation. A pair may be displayed in more than one scene. A user mayor may not have interacted with a pair. As used herein, in accordancewith some embodiments, a view is what a user sees through, e.g., an MRdisplay, which may include real-world objects, virtual objects, and ARinformation. A view may extend beyond a user's environment, for example,if the user is viewing an outdoor space extending to a distant horizon,rather than the user merely viewing nearby objects within a window-lessroom. As used herein, in accordance with some embodiments, persistenceis a period of time for defining how long a real-world object remainswithin a user's view. For some embodiments, prioritizing AR informationbased on persistence may be as simple as displaying AR information forhighest priority AR pairs first, with AR information for lower prioritypairs being displayed at a later time. For some embodiments,prioritizing AR information based on persistence may includehighlighting the objects, such as with an outline, or highlighting theAR information, such as with a different font size, font color, framing,and/or background color. Thus, the user's attention may be drawn toobjects of short persistence, to enable the user to benefit from ARinformation.

For some embodiments, prioritizing AR information based on persistencemay include presenting AR information first for objects having apredicted persistence shorter than a first threshold but longer than asecond threshold. The second threshold may prevent wasting a user'sattention for objects that departed the user's view too quickly. Thisenables a user to see AR information for objects that will disappearsoon, but yet remain in view for a sufficient length of time to exceedthe second threshold.

Systems and methods disclosed herein in accordance with some embodimentsmay focus on ongoing interactions that are at a risk of beinginterrupted and have been interrupted, in order to provide continuityfor those interactions. For example, an object may be of sufficientvalue or interest to the user for the user to begin to interact with it.However, the object may disappear from a user's view, due to themovement of that object or occlusion. In such a scenario, the user maynot have completed the planned interaction with the object and itsassociated AR information.

Systems and methods disclosed herein in accordance with some embodimentsmay detect that a relevant real-world object has moved out or will moveout of a user's view, determine that a user interaction has been or maybe interrupted as a result, and then compose an MR view that includes avirtual representation of that object and associates that virtualrepresentation with AR information about the real-world equivalent ofthe virtual object. In such embodiments, a user may benefit from ARinformation about an object even though the object is absent from theuser's view. Some embodiments may accomplish this by (1) identifying areal-world object that was present in a user's view but has only justmoved out of the view; (2) presenting a virtual object representing thatreal-world object; and (3) presenting AR information for the real-worldobject linked to the virtual object representation.

Systems and methods disclosed herein in accordance with some embodimentsmay present, through an MR interface, AR information associated with areal-world object that is absent from the user's view. For one exemplaryembodiment, such methods may comprise receiving a pair that is presentin the user's view; determining whether the user's interaction with thereceived pair has been interrupted by the real-world object becomingabsent from the user's view; and composing an MR view to enable the userto continue interacting with the pair by suitably placing an MRrepresentation of the pair into the view. As used herein, in accordancewith some embodiments, a present object is a real-world object that isvisible or otherwise inferred in a user's view. A present object mayinclude an object that is visually present, even if that object is notdirectly visible. As used herein, in accordance with some embodiments, apresent pair is a present real-world object and its associated ARinformation item. As used herein, in accordance with some embodiments,an absent object is a real-world object that is not present in a user'sview. As used herein, in accordance with some embodiments, an MRrepresentation of a pair is a representation suitable for an MRinterface in which the real-world object is represented as a virtualobject, and the real-world object's associated AR information ispresented by linking to or otherwise augmenting its correspondingvirtual object. For some embodiments, an MR representation of areal-world object may be composed only for those objects interactingwith a user, where that interaction has been interrupted for longer thana threshold time. For some embodiments, an MR representation of areal-world object may be composed only for those objects interactingwith a user, where associated AR information is not displayed to theuser recently within a threshold period of time.

Some embodiments may determine whether an object has been occluded orhas exited the scene. For occlusion, a virtual object representation maybe placed where the occluded object is calculated or otherwise expectedto be. Otherwise (such as via exiting the scene or, e.g., having anunpredicted disappearance), if a background object persists, therepresentation may be placed where the real-world object was seenrelative to the background object. If no background object persists, therepresentation may be placed where the object was last seen relative tothe user's view. As a result, in some embodiments, the MR view shouldplace the representation at a location that appears natural to the user.In some embodiments, the MR view may be composed by determining where toplace a virtual representation of a real-world object to enable aninteraction that seems natural to a user.

FIG. 2 is a message flow diagram 200 for an example method ofprioritizing AR information based on persistence of objects within theuser's view, according to some embodiments. As illustrated, one or morescene sensors 202 (for example, a camera, RGB-D sensor, light detectionand ranging (Lidar), and/or GPS) provides scene information 204 (e.g.,as a point cloud), including possibly location information, to an ARdisplay manager 206. AR display manager 206 receives scene information204 and sends combined scene and location information 208 to one or moreAR applications (AR Apps) 210. Although AR apps are described, it shouldbe understood that MR apps may also be used. AR apps 210 uses thereceived information 208 to produce a set of pairs 212 of identifiedreal-world objects and corresponding AR data. Each pair, in set of pairs212 includes information identifying an object (possibly using anidentification by location coordinates that includes an image of theobject and boundary coordinates) along with associated AR data thatprovides information about the object and may also describe how torender that information in an AR display. AR Apps 210 then provides setof pairs 212 to AR display manager 206.

According to the example, AR display manager 206 receives set of pairs212, and sends an identification of the objects 214 to a persistencedeterminer 216. Persistence determiner 216 computes persistence valuesfor each received object, for example, by determining the amount ofmotion of each object based on current and previous scenes. Persistencedeterminer 216 may calculate a prediction of when each object may moveout of view of the current scene, which may be based on an assumptionthat the current scene is a sufficiently accurate prediction of what theuser will see next. Persistence determiner 216 may send calculatedpersistence values 218 back to AR display manager 206. In someembodiments, AR display manager 206 gives greater priority to thepresentation of AR information relating to objects whose persistence iscalculated to be below a first threshold. In some embodiments, ARdisplay manager 206 gives greater priority to the presentation of ARinformation relating to objects whose persistence is calculated to bebelow a first threshold and above a second threshold. AR display manager206 then sends the selected objects and AR information 220 to an ARdisplay 222 for viewing by a user. The computation of persistence valueswill be described in more detail with later figures.

In some embodiments, AR display manager 206 operates on an ongoingbasis, performing the following actions: receiving scene information204, sending scene and location information 208 to AR applications 210,receiving object and AR information pairs 212, sending scene and objectinformation 214 to a persistence determiner 216, receiving real-worldobject persistence values 218, prioritizing AR information display, andsending display information 220 to an AR display 222. Scene informationmay be received from available sensors. Scene and location informationmay be sent to appropriate AR applications, such as subscribed ARapplications. An AR Display Manager may receive a list of pairs (asdescribed above) from each of multiple AR applications and compile amaster list of pairs.

It should be understood that, in some embodiments, AR display manager206 operates on a server remote from the UE, and in some embodiments, ARdisplay manager 206 is implemented within the UE. In some embodiments,AR display manager 206 and persistence determiner 216 may operate ondifferent nodes, and in some embodiments, AR display manager 206 andpersistence determiner 216 may operate within the same nodes. In someembodiments, AR display 222 operates as part of the same UE as one ormore scene sensors 202. It should be understood that, in someembodiments, at least some of AR apps 210 may be local and runningwithin the UE. Therefore, although the system illustrated in FIG. 2 isdescribed as a distributed system, in some embodiments, the method maybe performed within a single UE.

FIG. 3 is a message sequence diagram 300 for an example method ofprioritizing AR information based on persistence of objects within theuser's view, according to some embodiments. Scene sensor 202, which maybe part of a UE or an infrastructure device, sends scene information 204to AR display manager 206. AR display manager 206 may be a device, suchas a server, or a cloud-based module. AR display manager 206 sends sceneand location information 208 to an AR application (or informationservice) 210, which may be one of many AR apps. AR app 210 may be asoftware module resident in the cloud. AR app 210 may send pairs 212 ofobjects and AR information to AR display manager 206. AR display manager206 may send scene and objects 214 to persistence determiner 216. Forsome embodiments, persistence determiner 216, which may be a softwaremodule on a device or cloud-based module, determines 310 positions andvelocities of objects relative to a user's view and calculates 312persistence of objects (and associated persistence values). Persistencedeterminer 216 may send determined (calculated) persistence values 218for each object to AR display manager 206. For some embodiments, ARdisplay manager 206 selects 314 which pairs to display based on receivedpersistence values. AR display manager 206 sends scene informationaugmented with selected or prioritized AR information 220 to AR display222, which may be part of a UE.

In some embodiments, a method of calculating persistence values includesdetermining how long each object may persist within a user's view.Representative methods may employ computer vision (machine vision) todetermine velocity and visibility, perhaps one method for motiontracking and another method for visibility detection. Example motiontracking methods include, for example, a method described in: ChristophMertz, et al., Moving Object Detection with Laser Scanners, 30(1) J.Field Robotics 17-43 (2013), available athttps://www.ri.cmu.edu/pub_files/2013/1/rob21430.pdf. Example visibilitydetermination methods include, for example, a method described in SagiKatz, et al., Direct Visibility of Point Sets, 26:3 ACM Transactions onGraphics (SIGGRAPH) 24:1-24:11 (August 2007), available athttp://webee.technion.ac.i1/˜ayellet/Ps/KatzTalBasri.pdf.

Some embodiments for calculating (determining) persistence values mayuse velocity and visibility detection of an object to determine how longthat object will persist in a user's view based on direction, speed oftravel, and the boundaries of the received point cloud. Some embodimentsfor calculating persistence values of objects computes an object'scoordinates and velocity. For example, velocity may be calculated basedon a motion tracking method that uses laser rangefinder data (such asLidar data) as an input, and outputs an object's coordinates andvelocity. Sometimes, a straight line of motion may be determined as aset of points corresponding to an object's anticipated motion for ashort duration less than a prediction threshold (for assuming constantmotion). In such embodiments, historical motion is used to predictfuture motion, with an assumption that constant motion may be likely tocontinue for the threshold value of time.

For some embodiments, this threshold may be set to 30 seconds as adefault value, to be adjusted upon need, or, e.g., experience indicatingthat a different threshold may be preferable. In some embodiments, thisthreshold may be set based on the user's velocity (if determined). Astationary user may observe substantially the same scene for a longerperiod of time than a user moving in a car on a high-speed roadway.Points in an object's line of motion may be limited to those pointscorresponding to the object's change in position based on its velocityand the prediction time threshold. Points corresponding to an object'sline of motion may be inserted into a scene's point cloud, and thusdisplayed to the user. A visibility detection method may be used todetermine which points will be visible to a user. For some embodiments,the present scene may be used as an estimate of a future scene that theuser will view. For some embodiments, determining point visibility mayuse a method similar to one described by Katz, et al. (referencedabove). Some visibility detection methods may compute visible pointsbased on surface reconstruction, although this is not used by allmethods. For some embodiments, based on the results of the pointvisibility method, visibility may be determined for points correspondingto a line of motion. The first point in a line of motion to bedetermined to be invisible indicates where an object may be occluded bysomething else in a scene. This point may be identified as a point ofdisappearance. If no point in a line of motion is determined to beinvisible, the object is determined to not leave the scene (or thevisible part of the environment) within the prediction threshold time.For some embodiments, based on current position, point of disappearance,and velocity of an object, a persistence value of an object may becalculated.

Thus, the system tracks motion of a plurality of real-world objects,based on the tracking of the objects, predictions are made regarding howlong each object is likely to persist in the user's field of view, andthe display of the AR information is prioritized for those objectshaving the shortest predicted persistence, that is longer than someminimum threshold time.

FIGS. 4A-4B are a sequence of user views that show a basis for apersistence determination that may be used in some embodiments. FIG. 4Aillustrates a scene 400 a at time t=0 that is used as a reference todetermine motion of objects. For this example, the user's view includesa circle object 402, a triangle object 404, and a building 406. FIG. 4Bis a scene 400 b, same view, but at a later time t=T (the currentscene). For some embodiments, by comparing scene 400 a at time t=0 withscene 400 b at time t=T, a motion tracker module may determinecoordinates and motion for objects 402, 404, and 406. In this example,circle 402 is moving, while triangle 404 and building 406 remainstationary. In FIG. 4B, circle object 402 is at a new position 408 a,and predicted positions 408 b-408 f are also illustrated. Position 408 fmay be the final predicted position within the prediction thresholdtime. In some embodiments, ghost images may be inserted at predictedpositions 408 b-408 f, where circle object 402 is calculated to bemoving (if the motion of circle object 402 is constant and the remainderof scene 400 b remains unchanged). Some of the ghost images may beoccluded by building 406, and some embodiments may use this informationto determine a persistence prediction for circle object 402.

Some embodiments may use information regarding motion and predictedpersistence of circle 402 to prioritize showing AR information aboutcircle object 402 over showing AR information about triangle 404. Someembodiments may deprioritize showing AR information about circle object402 if it has already been displayed to the user. Some embodiments maydeprioritize showing AR information about circle object 402 if it willbecome occluded within a shorter threshold period of time. That is, someprioritization methods may include decisions such as: (a) Has ARinformation about the object been shown recently? If yes, do not displayAR information again. (b) Is the object about to disappear from viewwithin a first threshold time? (c) Is the object about to disappear fromview within a second threshold time, shorter than the first thresholdtime? (d) If yes to (c), do not display AR information. (e) If yes to(b) and no to (c), then prioritize display of AR information for theobject. Multiple prioritization levels may be used in (e). For example,some embodiments may assign a prioritization score, with higherprioritization scores typically being assigned to lower persistencepredictions. In some embodiments, prioritization scores may then begrouped into prioritization levels. The AR information may then bedisplayed according to prioritization level, with AR informationcorresponding to AR pairs having higher prioritization levels or scoresbeing displayed prior to AR information corresponding to AR pairs havinglower prioritization levels or scores. In some embodiments, ARinformation for two or more objects, within the same prioritizationlevel, may be displayed simultaneously. For some embodiments, detectionof a first object moving may trigger calculation of persistenceprediction values for stationary objects that may be occluded by themoving object, and thus some stationary object AR information may beprioritized.

FIG. 5 is an illustration of a user view 500 with AR information formultiple objects, as may be observed with some embodiments. In thisillustrated example, a user is visiting an archeological site, alongwith some celebrities. Object recognition techniques may be used todetect and recognize both animate and inanimate objects. A first person502 is walking quickly in a direction such that first person 502 maysoon disappear behind a building. A second person 506 is walking awayslowly, such that, within the prediction threshold time, second person506 will remain within view. A third person 510 is sitting, motionless.Other detected objects 514, 518, 522 and 526 are archeological objects,and are thus unlikely to move on their own; any motion of these objects514, 518, 522 and 526 within user view 500 would be caused by the user'sown motion or head motions. In this example, however, the user isstanding motionless, with a consistent direction of view.

Based on the predicted persistence values, AR information 504 regardingperson 502 may be prioritized over AR information for other detectedobjects. Prioritization may include displaying first, or, if displayingsimultaneously with other AR information, displaying with font size orcoloration (text and/or background) in a manner different thannon-prioritized AR information, such that it attracts a user's attentionmore readily. Because of the slower motion of person 506, but expectedcontinued persistence, AR information 508 about person 506 may beprioritized above AR information for stationary archeological objects514, 518, 522 and 526, but yet below AR information 504 about person502. Although person 510 is stationary for the time being, but yet is ahuman capable of motion rather than a stationary archeological object,AR information 512 about person 510 may be prioritized above ARinformation for stationary archeological objects 514, 518, 522 and 526,but yet below AR information 504 about person 502. AR information 512about person 510 may be prioritized similarly as AR information 508about person 506, or lower because person 506 is moving and person 510is not moving. In some embodiments, multiple prioritization levels maybe used, including predicted rapid disappearance, motion but nopredicted disappearance, no motion but an object capable of motion, andno motion and the object is of a type for which motion would not beexpected.

Mixed Reality Presentation of AR Information.

FIG. 6 is a message flow diagram 600 for an example method ofprioritizing AR information and handling interruptions for an MR system,according to some embodiments. As illustrated, one or more scene sensors202 (for example, a camera, RGB-D sensor, light detection and ranging(Lidar), and/or GPS) provides scene information 604 (e.g., as a pointcloud), including possibly location information, to an MR presentationmanager 606. MR presentation manager 606 receives scene information 604and sends combined scene and location information 608 to one or more ARapplications (AR Apps) 210. Although AR apps are described, it should beunderstood that MR apps may also be used. AR apps 210 uses the receivedinformation 608 to produce a set of pairs 612 of identified objects(real or virtual) and corresponding AR data. Each pair, in set of pairs612 includes information identifying an object (possibly using anidentification by location coordinates that includes an image of theobject and boundary coordinates) along with associated AR data thatprovides information about the object and may also describe how torender that information in an MR presentation. AR Apps 210 then providesset of pairs 612 to MR presentation manager 606.

MR presentation manager 606 receives set of pairs 612, and sends anidentification of the objects 614 to an interruption determiner 616.Additionally, one or more sensors 624 on a UE or user's person mayprovide information 626 about the user, e.g., a user's gaze andgestures, to interruption determiner 616. Interruption determiner 616determines which user interaction (e.g. with which object) wasinterrupted or may soon be interrupted on account of the object becomingabsent from a user's view. Interruption determiner 616 may calculate aprediction of when each object may move out of view of the currentscene, which may be based on an assumption that the current scene is asufficiently accurate prediction of what the user will see next.Interruption determiner 616 may send calculated interruption information618 back to MR presentation manager 606. In some embodiments, MRpresentation manager 606 gives greater priority to the presentation ARinformation and possibly virtual object presentation, based oninterruptions. MR presentation manager 606 composes a mixed reality view620 which may include virtual objects corresponding to occludedreal-world objects and associated AR information, and sends the data toan MR display 622. MR display 622 may be a WTRU, such as WTRU 102 b.

In some embodiments, MR presentation manager 606 operates on an ongoingbasis, performing the following actions: receiving scene information604, sending scene and location information 608 to AR applications 210,receiving object and AR information pairs 612, sending scene and objectinformation 614 to an interruption determiner 616, receivinginterruption information 618, composes a mixed reality view which mayinclude virtual objects corresponding to real-world objects andassociated AR information, and sending display information 620 to an MRdisplay 622.

FIG. 7 is a message sequence diagram 700 for an example method ofcreating an MR presentation, possibly including AR information, andhandling interruptions for an MR system, according to some embodiments.Scene sensor 202 sends initial scene information 604 a to MRpresentation manager 606. MR presentation manager 606 may be a device,such as a server, or a cloud-based module. One or more sensors 624 on aUE or user's person may provide initial information 626 a about theuser, e.g., a user's gaze and gestures, to interruption determiner 616.MR presentation manager 606 sends initial scene and location information608 a to AR application (or information service) 210, which may be oneof many AR apps. AR app 210 may be a software module resident in thecloud. AR app 210 may send initial pairs 612 a of objects and ARinformation to MR presentation manager 606. MR presentation manager 606sends initial scene and objects 614 a to interruption determiner 616.Interruption determiner 616 then determines 702 whether there is anyongoing interaction between a user and a pair.

The following may occur multiple times, as the MR session may be anongoing process: One or more sensors 624 may provide updated information626 b about the user, e.g., a user's gaze and gestures, to interruptiondeterminer 616. MR presentation manager 606 sends updated scene andlocation information 608 b to AR application 210. AR app 210 sendsupdated pairs 612 b of objects and AR information to MR presentationmanager 606. MR presentation manager 606 sends updated scene and objects614 b to interruption determiner 616. Interruption determiner 616determines 704 which user interaction (e.g. with which object) wasinterrupted or may soon be interrupted on account of the object becomingabsent from a user's view. Interruption determiner 616 may send thatdata as interruption information 618 to MR presentation manager 606. MRpresentation manager 606 composes 706 a mixed reality view which mayinclude virtual objects corresponding to occluded real-world objects andassociated AR information. MR presentation manager 606 sends sceneinformation augmented with selected AR information 620 to an MR display622, which may be part of a UE.

FIGS. 8A-8D are illustrations of MR user views for actual and virtualobjects under various conditions, as may be observed with someembodiments. FIG. 8A shows a scene 800 a at time t=0. In thisillustrated example, the user's view includes real-world circle object402, triangle object 404, and building 406. The pentagon shape is ARinformation window 804, pertaining to circle object 402. Circle object402 is moving, while triangle 404 and building 406 are stationary. FIG.8B shows a scene 800 b at time t=T1. Circle object 402 is hidden fromview (occluded) by building 406. The system determines that the userinteraction is not complete, for example the AR information window 804had not been displayed for sufficient time for the user to havecompleted reading the material. Thus, a virtual circle object 802 isdisplayed (e.g., in some embodiments, based on an expected position ofwhere the circle object 402 would be if not for being hidden from viewby building 406), along with the associated AR information window 804.

FIG. 8C is a scene 800 c at time t=T2. Real-world circle object 402re-emerges into view. The interruption determiner and MR presentationmanager together determine that (a) the user interaction is not over(more time is needed for the user to read AR information window 804),and (b) since real-world circle object 402 is in view, virtual circleobject 802 is no longer needed. FIG. 8D is a scene 800 d at time t=T3.The circle has moved out of view, not behind another object but out ofthe view of the camera. The interruption determiner and MR presentationmanager together determine that (a) the user interaction is still notyet over (more time is needed for the user to read AR information window804), and (b) since real-world circle object 402 has disappeared fromview, virtual circle object 802 is now needed again. Virtual circleobject 802 is placed, e.g., nearby where real-world circle object 402exited view, since that would be an intuitive location for a user tolook for AR information window 804. In this way, despite occlusion anddisappearance, user interaction continued without interruption, givingcontinuity to the user's experience. In some embodiments, the virtualcircle object 802 may be still be triggered by virtual circle object 802disappearing from view but virtual circle object 802 may be placedelsewhere, e.g., further away or elsewhere entirely, from wherereal-world circle object 402 exited view.

Determining interruption of a user's interaction with the received pairmay depend upon the nature of the interaction. One method may be to usedetection of where a user's eyes have settled on an AR informationwindow, and comparing those regions with the regions containing text.Various techniques may be used to track user gaze, as suitable. FIGS.9A-9D are illustrations showing example locations of user gazes withinan example AR information text window, for example the pentagon ARinformation window 804 of FIGS. 8A-8D.

The dashed circles with hashed shading in FIGS. 9A to 9D indicate wherethe user's gaze has settled. FIG. 9A shows a pentagon-shaped ARinformation window 902 upon which AR information may be presented to auser. The detected gaze locations 904 a-904 b cover only a small portionof window 902, indicating that, if the user interaction stops, therewill be an interruption. FIG. 9B shows a similar scenario in which auser has not completed an interaction, although using text block 906,rather than information window 902. The detected gaze locations 908a-908 e cover only a small portion of text block 906, indicating thatthe user has not completed reading all of the text. For FIG. 9C, thedetected gaze locations 910 a-9101 indicate that the user has readnearly all of the text. Thus, if the interaction stops, there may not bean interruption, or the interruption will not be significant. For FIG.9D, the detected gaze locations 912 a-912 e indicate that the user hasread significant portions (here determined to be the starting fragmentsof each line), suggesting that a user is speeding through the text andindicating that there is no interruption.

Other methods for determining interruption of a user's interaction withAR information may also be used, in some embodiments. An exemplary“short duration” interaction may be specified as one second or less. Ifa user's interaction with a pair (object and AR information) involvesinteracting with a real-world object in the pair and if the user's gazesettled upon any part of the real-world object within a short durationprior to when the real-world object becomes absent, the user'sinteraction with the real-world object may be determined to beinterrupted. If the user's interaction with the pair involvesinteracting with the associated AR information, the method may useattributes of the AR information in determining if an interruption of acontinuing user action has occurred.

For example, in scenarios in which AR information item has a spatialextent (e.g., as in a picture or a text snippet being displayed) then,if for a short duration prior to the moment when the real-world objectbecomes absent, the user's gaze had settled somewhere on that ARinformation without covering nearly the full spatial extent, then theuser's interaction may be determined to have been interrupted. Thiswould be the scenario shown in FIG. 9B, if, at the time the user's gazehad reached gaze location 908 e, the object disappeared from view.

If AR information that has a temporal extent (e.g., as in an audio clipor a video clip), and the entire AR information had not been played backwhen the real-world object becomes absent, the user's interaction may bedetermined to be interrupted. For a video clip, if the user's gaze isupon the video clip a short duration prior to a real-world objectbecoming absent but the video clip has not terminated, the user'sinteraction may be determined to be interrupted. For an audio clip, if ashort duration prior to the real-world object becoming absent, theuser's gaze is upon some part of the scene related to the object whilethe audio is playing but the audio clip has not terminated, the user'sinteraction may be determined to be interrupted. The case where a user'sgaze is upon a real-world object while the audio plays is discussedbelow.

Creating a mixed reality view enables a user to continue interactingwith a pair by placing a mixed reality representation of the pair in ascene. Systems and methods described herein in accordance with someembodiments may use the background to anchor the scene so that thevirtual representation of the real-world object may be placed in alocation natural to the user. For one embodiment, this method may beused where the scene has completely changed and no constant backgroundis detected. To determine the coordinates and a velocity of the object,a motion tracking method may be used. An exemplary tracking method thatmay be used is a method that uses input rangefinder data and outputscoordinates and a velocity of an object as described in Christoph Mertz,et al., Moving Object Detection with Laser Scanners, 30(1) J. FieldRobotics 17-43 (2013), available athttps://www.ri.cmu.edu/pub_files/2013/1/rob21430.pdf.

If the real-world object is not visible in the current scene and anotherobject is detected along the ray from the user's eyes to the predictedcoordinates of the real-world object, the real-world object may bedetermined to be occluded by the first object on the ray going from theuser's eyes to the real-world object. There may be multiple occludingobjects for a partially overlapping effect.

FIGS. 10A-10C are illustrations of MR user views with a moving object,under various scenarios, as may be observed in some embodiments. Toindicate the 3D nature of the layouts, coordinate axis, beginning fromorigin 1002 is shown, where x=0, y=0, and z=0. The x-axis 1004, y-axis1006 and negative z-axis 1008 are illustrated for reference. In theexample illustrated by FIG. 10A, a real-world circle object 1010 a inscene 1000 a becomes occluded by building 406 as real-world circleobject 1010 a moves along motion vector (velocity) 1012 a from itsinitial position <xp, yp, zp> to position 1010 b at coordinates <xc, yc,zc>. Sight vector 1014 a indicates that real-world circle object 1010 ais visible, initially. Sight vector 1016 a indicates that real-worldcircle object 1010 a is occluded when in position 1010 b, and sointerruption determiner and MR presentation manager together maydetermine that a virtual representation 1018 of circle object 1010 a isneeded and should be placed along sight vector 1016 a. This location isspecified by the coordinates <xo, yo, zo>. If circle object 1010 acontinues moving and reappears on the right side of building 406 asviewed by the user, virtual representation 1018 may no longer be used.

In the example illustrated by FIG. 10B, circle object 1010 a exits scene1000 b by moving along motion vector 1012 b from its initial position<xp, yp, zp> to position 1010 b at coordinates <xc, yc, zc>. Sightvector 1014 a indicates that circle object 1010 a is visible, initially.Sight vector 1016 b indicates that circle object 1010 a is absent fromscene 1000 b, and so interruption determiner and MR presentation managertogether may determine that a virtual representation (not shown) may beneeded nearby the location where circle object 1010 a exited scene 1000b.

In some embodiments, a virtual representation of a real-world object,for example virtual representation 1018 for circle object 1010 a, may beplaced in the scene at coordinates where the relationship between themissing object and some objects in the background remains fixed. A usermay find placing the missing object at a location fixed to backgroundobjects more natural for the user to continue interacting with thereal-world object or its associated AR information without losingcontinuity. For example, if a car is seen as to the right of a lightpole, an image of the missing car may also be displayed to the right ofthat light pole.

In some situations, however, a real-world object may move in anunpredicted manner, and may vanish in a surprise (unpredicted) absence.In the example illustrated by FIG. 10C, circle object 1010 a startedwithin scene 1000 c its initial position <xp, yp, zp>, and a motionvector 1012 c was calculated for it, predicting possible occlusion bybuilding 406 and reemergence at position 1010 b at coordinates <xc, yc,zc>. Position 1010 b at coordinates <xc, yc, zc> should be visible,based on sight vector 1016 c. However, circle object 1010 a is absentfrom scene 1000 c. Thus, interruption determiner and MR presentationmanager together may determine that a virtual representation (not shown)is needed and should be placed along sight vector 1016 c, at or nearbythe position 1010 b where circle object 1010 a would have been predictedto remerge. This location 1010 b is specified by the predictedcoordinates <xc, yc, zc>. In some embodiments, if the circle object 1010a continues moving and eventually reappears on the right side ofbuilding 406 as viewed by the user and (somewhat) as originallypredicted, virtual representation 1018 may, e.g., no longer be used andthe scene 1000 c will revert to showing the circle object 1010 a, in,e.g., its now current location.

FIGS. 11A-11C are illustrations of MR user views involving calculationof displacement or placement of objects, as may be observed in someembodiments. In FIG. 11A a background object 1112 is shown anchored atlocation <xb, yb, zb> in scene 1100 a. A displacement vector 1114 a iscalculated as the difference between the coordinates <xp, yp, zp> of anobject 1110 and the location of background object 1112. Thisdisplacement is indicated by <dx, dy, dz>. There is a sight vector 1116a from origin 1002 to background object 1112, and a sight vector 1118 afrom origin 1002 to object 1110. The sight vectors 1116 a and 1118 a maybe used in determining coordinates, in some embodiments.

FIG. 11B depicts the scenario in which the user has changed positionsand is now viewing scene 1100 b. Object 1110 is no longer visible,perhaps due to occlusion by building 406, or perhaps because object 1110moved out of view. Thus, a virtual representation 1120, which may be a3D model or 2D image, of object 1110 is to be used for user interaction,such as display of AR information. In this example, virtualrepresentation 1120 will be placed at an anchoring location of <xn+dx,yn+dy, zn+dz>. This location is based on displacement vector 1114 a frombackground object 1112, as was calculated for scene 1100 a (of FIG.11A). The calculation of <dx, dy, dz> is shown in FIG. 11A.

There is a sight vector 1116 b from origin 1002 to background object1112, and a sight vector 1118 a from origin 1002 to representation 1120.The sight vectors 1116 b and 1118 b may be used in determiningcoordinates, in some embodiments.

FIG. 11C depicts the scenario in which the user has changed positionsagain and is now viewing scene 1100 c. None of object 1110, backgroundobject 1112, and building 406 are visible, although a new backgroundobject 1124 is visible. Thus, there is no anchor point for adisplacement vector to define the display location of representation1120 in scene 1100 c. Therefore, in some embodiments, representation1120 will be placed according to sight vector 1118 b, so that it is inthe same location with respect to the frame of reference asrepresentation 1120 had been placed in scene 1100 b, as shown in FIG.11B.

In some embodiments, AR information associated with an object may bepresented to the user in a way that the user associates the ARinformation with an object of the real-world. One exemplary embodimentof such a presentation uses the same modality as may have been used forother AR information in the current settings, so that the user readilyassociates the presented AR information with the real-world object whoseimage is being shown. For some embodiments, if the AR information hasspatial extent, e.g., a picture or a text snippet, all of the ARinformation continues to be shown. For some embodiments, if the ARinformation has temporal extent, e.g., an audio or video clip, the ARinformation is presented from where the AR information was lastinterrupted.

In some embodiments, a determination that the interaction has not endeddepends upon the content of the AR information, e.g., if a video clip is30 seconds long. In some embodiments, determination that the interactionhas not ended depends upon the observations of the user, e.g., theuser's gaze. In some embodiments, the image for the real-world objectthat has disappeared from view without exiting the scene is displayed atits predicted coordinates instead of relating the image to thebackground. In some such embodiments, doing so may avoid having tocompute the background of the scene.

In some embodiments, a video of a real-world object as it traveledacross part of the user's view may be displayed with the appearance ofthe real-world object transformed to indicate that it is not present. Avideo representation may be used if a real-world object is traveling ata steady velocity and is occluded by something else in the user's view.In some embodiments, a slow-motion video of a real-world object isdisplayed as the real-world object traveled across part of the user'sview. The slow-motion video modality may be used if the real-worldobject departs from the edge of the user's view.

FIGS. 12A-12B are AR/MR user views for a scenario in which an object isobscured, as may be observed with some embodiments. FIG. 12A illustratesan AR view for a video on creating a mixed drink in an exemplary usecase. A user encounters a particular mixed drink 1202 on a counter. AnAR presentation manager outlines mixed drink 1202 and displays ARinformation 1204, in the form of a video clip on making the drink, asscene 1200 a. The video clip begins playing in the user's MR display.

FIG. 12B illustrates the situation in which an occluding object 1206 (aperson) obscures the mixed drink 1202 prior to the completion of thevideo in AR information 1204. Therefore, a virtual representation 1208is placed at the last identified location of mixed drink 1202, and thevideo clip in AR information 1204 continues to play.

FIGS. 13A-13B are AR/MR user views for a scenario in which an objectmoves out of view, as may be observed with some embodiments. Asillustrated in FIG. 13A, an AR app using object detection andrecognition algorithms has identified a particular model of car 1304 inscene 1300 a, and has displayed AR information as text in AR informationwindow 1304. Additionally, car 1302 is outlined to highlight it for theuser's attention. In this example car 1302 is stopped at a red trafficlight. An image is captured of car 1302 for later use as a virtualrepresentation, in the event that car 1302 disappears from view prior tothe user having time to finish reading the AR information text.

FIG. 13B illustrates the interruption scenario in scene 1300 b. Thetraffic light had changed, and car 1302 sped away, out of view. Usingsystems and methods disclosed herein in accordance with someembodiments, the user's interaction with car 1302 may be determined tobe interrupted. Therefore, according to the example. an image 1306 ofcar 1302 is placed in the user's MR display, and AR information window1304 is associated with image 1306, since real-world car 1302 is absent.Various techniques for locating a virtual representation or image may beused in differing embodiments. For example, an image may be placed inthe same relative location within a scene relative to a background, orin the same viewing direction.

Heads-Up Vehicle Display.

FIGS. 14A-14E are illustrations of a heads-up AR display in variousscenarios, as may be observed with some embodiments. Some embodimentsmay be implemented as a heads-up display in an automobile, to assist adriver or passenger (either being the user) with a view out of the car'swindshield. In some embodiments, images may be captured with a forwardfacing camera on or inside the car. FIG. 14A shows a view 1400 a out ofa car windshield, at a time t=0, with two signs 1402 and 1404 in view.No AR information is yet displayed.

FIG. 14B shows a view 1400 b out of the car windshield, at a slightlylater time t=T1, when sign 1404 is still in view, but sign 1402 has beenpassed and is no longer in view. AR information 1406, regarding sign1404 is displayed on a heads-up display for the user.

FIG. 14C shows a view 1400 c out of the car windshield, at the slightlylater time t=T1, when sign 1404 is still in view, but sign 1402 has beenpassed and is no longer in view. Rather than displaying AR informationas text, an icon 1408, related to sign 1404 is displayed on a heads-updisplay for the user. In some embodiments, if the user taps the icon inthe heads-up display, more information related to sign 1404 may beprovided. FIG. 14D shows a view 1400 d out of the car windshield, at theslightly later time t=T1, similar to view 1400 c of FIG. 14C, but with adifferent example location for icon 1408. In some embodiments, if a userhas gazed at a recognized sign, e.g., for less than a first thresholdtime period but greater than a second threshold time period, ARinformation or an icon related to the sign will be displayed.

FIG. 14E shows a view 1400 e out of the car windshield, at a later timet=T2, when both sign 1402 and sign 1404 have been passed and are nolonger in view. Two menus, 1410 and 1412 are presented that may berelated to sign 1402 and/or sign 1404. For some embodiments, ratherthan, e.g., AR information being displayed in a textual format, menusmay be displayed. For some embodiments, AR information related to a signis displayed if a user has interacted with a user interface element(such as a menu). For some embodiments, the AR information or userinterface item related to a sign is displayed if a user has gazed at thesign for, e.g., more than a first threshold but less than a secondthreshold. For some embodiments, AR information related to a sign may bedisplayed on a heads-up display at a fixed displacement from abackground object. For some embodiment, AR information related to a signmay be displayed at a fixed displacement from a background object if thesign has disappeared from view of the heads-up display.

Exemplary Applications

In some embodiments, determining persistence values may includedetermining how long each object may persist within a user's view, ordetermining an object's velocity and an object's visibility. Determiningan object's velocity may include performing motion tracking of theobject. Determining persistence values may include determining how longan object will persist in a user's view based on the object'scoordinates, the object's velocity, and boundaries of the received sceneinformation. In some embodiments, determining persistence values may usea straight line of motion model, or using historical motion to predictfuture motion. Persistence values may be determined up to thresholdprediction time limit, which in some embodiments may be 30 seconds. Insome embodiments, determining persistence values may include predictingpoints in a line of motion of an object and/or displaying the predictedpoints in the line of motion of the object. Visibility may be determinedfor the predicted points in the line of motion of the object, as part ofdetermining persistence values. In some embodiments, determiningpersistence values may include determining occlusion of an object. Insome embodiments, determining persistence for objects in the pluralityof AR pairs may include sending the plurality of AR pairs to apersistence determiner; and receiving the determined persistence forobjects in the plurality of AR pairs.

In some embodiments, prioritizing display of AR information may includepresenting AR information first for objects having a predictedpersistence shorter than a first threshold. In some embodiments,prioritizing display of AR information may include presenting ARinformation first for objects having a predicted persistence shorterthan a first threshold but longer than a second threshold. Prioritizingdisplay of AR information may include determining whether AR informationfor an object has been displayed; and responsive to determining that ARinformation for the object has been displayed, not displaying the ARinformation for the object again. Prioritizing display of AR informationmay include assigning a prioritization score to an AR pair; anddisplaying AR information corresponding to pairs having higherprioritization scores prior to displaying AR information correspondingto pairs having lower prioritization scores. Prioritizing display of ARinformation may include highlighting a display of the AR informationwith a different font size or color than AR information that is notprioritized. Some embodiments of the method may further include:responsive to receiving the scene information, sending the sceneinformation to an information service, and wherein receiving a pluralityof AR pairs includes receiving the plurality of AR pairs from theinformation service. Receiving scene information may include sensingscene information with a scene sensor. And in some embodiments, sceneinformation may include a point cloud.

In some embodiments, a method of prioritizing AR information based onobject persistence within a user's view may include tracking motion of aplurality of real-world objects; based on the tracking of the objects,predicting how long each object is likely to persist in the user's fieldof view; and prioritizing display of the AR information for thoseobjects having shorter predicted persistence, wherein prioritizingdisplay of the AR information comprises displaying first.

In some embodiments, a system may include a processor; and anon-transitory computer-readable medium storing instructions that areoperative, when executed by the processor, to perform the methodsdisclosed herein. A system may further include a display and a scenesensor.

Note that various hardware elements of one or more of the describedembodiments are referred to as “modules” that carry out (i.e., perform,execute, and the like) various functions that are described herein inconnection with the respective modules. As used herein, a moduleincludes hardware (e.g., one or more processors, one or moremicroprocessors, one or more microcontrollers, one or more microchips,one or more application-specific integrated circuits (ASICs), one ormore field programmable gate arrays (FPGAs), one or more memory devices)deemed suitable by those of skill in the relevant art for a givenimplementation. Each described module may also include instructionsexecutable for carrying out the one or more functions described as beingcarried out by the respective module, and it is noted that thoseinstructions could take the form of or include hardware (i.e.,hardwired) instructions, firmware instructions, software instructions,and/or the like, and may be stored in any suitable non-transitorycomputer-readable medium or media, such as commonly referred to as RAM,ROM, etc.

Although features and elements are described above in particularcombinations, one of ordinary skill in the art will appreciate that eachfeature or element can be used alone or in any combination with theother features and elements. In addition, the methods described hereinmay be implemented in a computer program, software, or firmwareincorporated in a computer-readable medium for execution by a computeror processor. Examples of computer-readable storage media include, butare not limited to, a read only memory (ROM), a random access memory(RAM), a register, cache memory, semiconductor memory devices, magneticmedia such as internal hard disks and removable disks, magneto-opticalmedia, and optical media such as CD-ROM disks, and digital versatiledisks (DVDs). A processor in association with software may be used toimplement a radio frequency transceiver for use in a WTRU, UE, terminal,base station, RNC, or any host computer.

What is claimed:
 1. A method of prioritizing augmented reality (AR)information based on object persistence within a user's field of view,the method comprising: tracking motion of a plurality of real-worldobjects; based on tracking the motion of the plurality of real-worldobjects, predicting how long each object of the plurality of real-worldobjects is likely to persist within the user's field of view; andprioritizing display of the AR information for those real-world objectshaving shorter predicted persistence, wherein prioritizing display ofthe AR information comprises displaying AR information for thosereal-world objects having shorter predicted persistence first.
 2. Themethod of claim 1, further comprising: receiving scene information; andsending the scene information augmented with prioritized AR informationto a display.
 3. The method of claim 2, further comprising: receiving aplurality of augmented reality (AR) pairs corresponding to the pluralityof real-world objects, each AR pair comprising an identification of anobject of the plurality of real-world objects and AR informationassociated with the identified object, wherein predicting how long eachobject of the plurality of real-world objects is likely to persist inthe user's field of view comprises determining persistence values forthe plurality of real-world objects in the plurality of AR pairs, andwherein prioritizing display of the AR information further comprisesprioritizing display of the AR information for the real-world objectsbased on the determined persistence values for the correspondingreal-world objects.
 4. The method of claim 1, wherein predicting howlong each object of the plurality of real-world objects is likely topersist within the user's field of view comprises determining theobject's velocity and the object's visibility.
 5. The method of claim 2,wherein predicting how long each object of the plurality of real-worldobjects is likely to persist in the user's field of view is based on theobject's coordinates, the object's velocity, and boundaries of thereceived scene information.
 6. The method of claim 1, wherein predictinghow long each object of the plurality of real-world objects is likely topersist in the user's field of view comprises using a straight line ofmotion model.
 7. The method of claim 1, wherein predicting how long eachobject of the plurality of real-world objects is likely to persist inthe user's field of view comprises using historical motion to predictfuture motion for each object.
 8. The method of claim 1, whereinpredicting how long each object of the plurality of real-world objectsis likely to persist in the user's field of view comprises predictingfuture motion for each object up to a threshold prediction time limit.9. The method of claim 1, wherein: predicting how long each object ofthe plurality of real-world objects is likely to persist in the user'sfield of view comprises: predicting points in a line of motion of eachobject, wherein the method further comprises displaying the predictedpoints in the line of motion of the object.
 10. The method of claim 1,wherein predicting how long each object of the plurality of real-worldobjects is likely to persist in the user's field of view comprisesdetermining visibility for the predicted points in the line of motion ofthe object.
 11. The method of claim 1, wherein predicting how long eachobject of the plurality of real-world objects is likely to persist inthe user's field of view further comprises: determining occlusion ofeach object of the plurality of objects.
 12. The method of claim 1,wherein displaying the AR information for those real-world objectshaving shorter predicted persistence first comprises: presenting the ARinformation first for objects having a predicted persistence shorterthan a first threshold.
 13. The method of claim 1, wherein displayingthe AR information for those real-world objects having shorter predictedpersistence first comprises: presenting the AR information first forobjects having a predicted persistence shorter than a first thresholdbut longer than a second threshold.
 14. The method of claim 1, whereinprioritizing display of the AR information further comprises:determining whether AR information for an object of the plurality ofreal-world objects has been displayed; and responsive to determiningthat AR information for the object has been displayed, not displayingthe AR information for the object again.
 15. The method of claim 3,wherein prioritizing display of the AR information further comprises:assigning a prioritization score to each AR pair of the plurality of ARpairs; and displaying AR information corresponding to AR pairs havinghigher prioritization scores prior to displaying AR informationcorresponding to AR pairs having lower prioritization scores.
 16. Themethod of claim 3, further comprising: responsive to receiving the sceneinformation, sending the scene information to an information service,wherein receiving a plurality of AR pairs comprises receiving theplurality of AR pairs from the information service.
 17. The method ofclaim 2, wherein receiving scene information comprises sensing sceneinformation with a scene sensor.
 18. A system comprising: a processor;and a non-transitory computer-readable medium storing instructions thatare operative, when executed by the processor, to cause the system to:track motion of a plurality of real-world objects; based on tracking themotion of the plurality of real-world objects, predict how long eachobject of the plurality of real-world objects is likely to persistwithin a user's field of view; and prioritize display of augmentedreality (AR) information for those real-world objects having shorterpredicted persistence, wherein prioritizing display of the ARinformation comprises displaying the AR information for those real-worldobjects having shorter predicted persistence first.
 19. The system ofclaim 18, further comprising: a scene sensor; and a display.